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1.  Introduction 


This  report  discusses  how  human  factors  engineering  (HFE)  is  a  critical  element  in  the  process 
of  sensor  fusion.  For  this  report,  the  definition  of  sensor  fusion  is  the  one  created  by  the  Joint 
Directors  of  Laboratories  (JDL)1.  With  the  JDL  definition  and  models,  we  review  each  sensor 
fusion  process  step  from  an  HFE  perspective,  we  discuss  how  human  factors  variables  play  a 
significant  role  for  each  element  of  that  process,  and  we  discuss  how  enhancing  the  role  of  HFE 
in  the  sensor  fusion  process  will  result  in  a  more  effective  human-machine  system.  Examples  to 
illustrate  these  issues  often  refer  to  the  military  domain,  military  aviation,  military  intelligence, 
automated  systems,  or  computer  operations. 

1.1  Background 

1.1.1  Basic  Concepts 

Sensor  fusion  is  the  process  of  collecting  data,  combining  those  data  through  a  variety  of  methods, 
with  a  variety  of  sensing  technologies,  and  presenting  those  data  as  an  integrated  product  to  a 
machine  or  a  human.  The  basic  advantage  to  fusing  data  is  that  (a)  it  should  be  faster  and  more 
effective  in  presentation  than  if  the  data  were  not  fused  and  (b)  combining  data  results  in  a  richer, 
more  intelligible  output  that  is  functionally  greater  than  the  sum  of  its  parts.  The  following  defini¬ 
tion  emphasizes  the  concept  of  fusion  from  a  cognitive  perspective,  regardless  of  the  type  of  sensor 
system  involved: 

Fusion  pools  multiple  bodies  of  evidence  to  produce  a  single  body  of  evidence 
that  emphasizes  points  of  agreement  and  de-emphasizes  points  of  disagreement 
(SRI2,  2006). 

This  definition  of  fusion  emphasizes  that  the  process  of  fusing  data  is  essentially  like  a  logical 
argument,  in  which  principles,  facts,  and  rules  all  combine  to  form  and  then  persuade  a  conclusion. 
It  is  analogous  to  the  testing  of  a  hypothesis  in  the  scientific  process  and  is  called  an  evidential 
operation  (SRI,  2006);  however,  fusion  is  also  selective  in  its  processing  of  information  and  may 
not  always  follow  a  strict  a  priori  logical  approach.  This  is  evident  in  the  consideration  of  human- 
based  fusion  of  information  where  heuristics  (rules  of  thumb)  or  recognition-primed  decisions  may 
sort  information  in  ways  that  machines  are  not  currently  capable  of  doing.  In  principle,  fusion  may 
be  accomplished  by  human  or  machine  systems. 

Most  of  the  discussion  in  this  report  relies  on  the  fact  that  machines  have  electro-mechanical 
sensors  and  computer-based  cognition,  whereas  humans  have  the  five  senses  plus  human  cogni¬ 
tion.  Humans  combine  the  input  of  their  senses  into  a  cohesive  product  through  cognition,  which 
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then  can  result  in  actions.  As  a  discussion  point,  it  is  posited  that  humans  operate  in  ways  similar 
to  machines,  from  the  standpoint  of  combining  input  data  and  information  into  an  output  action. 
This  system’s  view  recognizes  differences  as  well  as  similarities  between  machines  and  humans  in 
the  acquisition  and  processing  of  data,  as  well  as  the  final  output  products  achieved  by  both.  Some 
of  these  similarities  and  differences  are  discussed  in  this  report. 

Traditional  sensor  fusion  often  involved  detailed  mathematical  algorithms  that  controlled  the 
processing  of  information  to  reach  a  complex  goal  such  as  detecting,  classifying,  and  identifying 
a  target  (Ceruti,  2004).  However,  for  the  purposes  of  this  discussion,  the  interaction  of  human 
cognitive  processes  in  creating  and  using  infonnation  is  the  focus  of  the  discussion.  For  example, 
in  the  aviation  environment,  some  sensors  provide  system  status  infonnation,  others  provide 
mission  information,  while  still  others  provide  communications  linkages,  weapons  status,  or 
current  enemy  threat  status.  A  similar  process  is  followed  in  the  electronic  warfare  environment 
where  “multiple  levels  of  abstraction,  multiple  sensors  and  various  evidential  operations  all 
combine  to  predict  changes  in  observables”  (SRI,  2006).  The  original  goal  for  electronic  warfare 
and  aerospace  system’s  sensor  fusion  was  to  combine  all  these  sources  of  data  into  a  cohesive 
whole  from  which  a  machine  or  human  operator  could  draw  in  order  to  make  more  effective 
operational  decisions  (Bowman  &  Murphy,  1980).  This  approach  for  aerospace  was  in  marked 
contrast  to  earlier  aerospace  systems  where  routines  were  fully  automated  (programmed  or  “hard 
wired”  as  in  an  early  ballistic  missile),  and  it  was  also  in  contrast  to  systems  where  federated  or 
isolated  sensors,  acting  as  single  sources  and  providing  information  for  a  single  purpose,  were 
usually  monitored  by  a  human  operator  (as  in  an  early  aircraft  cockpit  or  an  intelligence  analyst 
monitoring  one  radio  channel).  Now,  on  advanced  aircraft  such  as  the  F-22,  the  pilot  manipulates 
the  throttles  to  provide  a  thrust  request  from  the  central  processing  computer  on  the  aircraft,  and 
the  computer  in  turn  determines  if  that  request  can  be  complied  with,  based  on  fused  input  from 
sensors  that  monitor  aircraft  and  external  parameters.  Those  variables  are  in  turn  fused  by  a  main¬ 
frame  computer  and  the  thrust  request  is  completed  or  mitigated,  which  is  very  different  from 
traditional  manual  control  of  that  process.  Likewise,  modem  electronic  warfare  systems  may 
include  sensor  feeds  from  multiple  sources  that  are  combined  on  a  single  display  fonnat  into  a  new 
image  reflecting  the  qualities  of  all  the  sensors  but  not  resembling  any  one  sensor. 

1.1.2  Sensor  Fusion  as  a  Process 

In  terms  of  human-computer  interaction,  current  sensor  fusion  systems  provide  a  cohesive 
picture,  often  using  graphic  and  auditory  technologies.  Display  screens,  headsets,  and  auditory 
warnings  are  all  examples  of  the  final  delivery  means  from  which  raw  sensor  data  are  fused  into 
information  and  then  channeled  to  the  system  operator  in  ways  they  can  comprehend  accurately 
and  quickly. 

1.1.3  Sensor  Fusion  Defined 

The  JDL’s  (White,  1988)  Sub-panel  on  Data  Fusion  has  defined  data  fusion  as 
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. .  .a  process  dealing  with  the  association,  correlation,  and  combination  of  data  and 
information  from  single  and  multiple  sources  to  achieve  refined  position  and  identity 
estimates,  and  complete  and  timely  assessments  of  situations  and  threats,  and  their 
significance.  The  process  is  characterized  by  continuous  refinements  of  its  estimates  and 
assessments,  and  the  evaluation  of  the  need  for  additional  sources,  or  modification  of  the 
process  itself,  to  achieve  improved  results... 

This  definition  provides  guidance  that  describes  a  closed  loop  learning  process  that  iteratively 
refines  an  assessment  of  a  situation,  much  like  a  sensor-fused  guidance  system  supplying 
navigation  input  to  a  missile.  Note  that  the  JDL  definition  provides  a  process  statement  that  uses 
human  information  processing  techniques  of  association,  correlation,  and  a  combination  of  data 
and  information  from  numerous  sources  (sensor  fusion)  to  identify  the  goals  refine  position  and 
identity  estimates  and  to  quickly  and  iteratively  make  assessments  of  situations  and  threats  while 
using  a  closed  loop  approach  to  achieve  improved  results.  What  is  missing  from  this  definition  is 
the  focus  on  how  a  human  can  perform  this  process,  particularly  in  an  environment  where 
automated  systems  are  providing  the  bulk  of  the  information  and  the  human  is  under  the  dynamic 
conditions  of  multiple  tasking. 

A  review  of  the  data  fusion  model  as  proposed  by  the  JDL  describes  the  data  fusion  domain  as  one 
of  parallel  processing  with  sources  such  as  “national,  distributed,  and  local”  providing  intelligence, 
electronic  warfare,  sonar,  radar,  or  other  types  of  information  sources  as  outside  the  central  block 
that  describes  the  five  levels  of  processing  (see  figures  1  and  2).  In  addition,  the  human-computer 
interface  is  seen  as  outside  the  central  data  fusion  domain  (it  is  shown  to  the  right  side  of  the 
diagram).  The  levels  are  described  in  figure  1  and  are  shown  in  a  model  format  in  figure  2.  The 
JDL  model  shown  in  figure  2  also  includes  another  step  which  is  user  refinement  (an  element  of  a 
closed  loop  process).  The  two  figures  also  show  the  changes  that  have  occurred  in  the  model  over 
time. 


The  JDL  Model  (1987-91) 
and  the  draft  revised  model  (1997) 


Level  0  — 
Level  1  — 

Level  2  — 
Level  3  — 

Level  4  — 


Sub-Object  Data  Association  and  Estimation:  Pixel/signal  level  data  association  and 
characterization 

Object  Refinement:  Observation-to-track  association,  continuous  state 
estimation  (e.g.  kinematics)  and  discrete  state  estimation  (e.g.  target  type  and 

ID)  and  prediction 

Situation  Refinement:  Object  clustering  and  relational  analysis,  to  include 
force  structure  and  cross  force  relations,  communications,  physical  context,  etc. 
Significance  Estimation  [Threat  Refinement]:  Threat  intent  estimation, 
event  prediction],  consequence  prediction,  susceptibility  and  vulnerability 
assessment 

Process  Refinement:  Adaptive  search  and  processing  (an  element  of  resource 
management) 


Figure  1.  The  JDL  model,  1987-91  with  1997  revisions  (Steinberg,  Bowman,  &  White,  1999). 
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JDL-User  Model 


Figure  2.  Revised  JDL  user  model,  2003-2006  (Blasch  &  Plano,  2003). 


2.  Purpose  of  Report 


The  purpose  of  this  report  is  to  provide  a  general  review  of  the  sensor  fusion  process  as  defined 
by  the  JDL  from  the  perspective  of  selected  human  engineering  principles. 


3.  Method 


3.1  Human  Engineering  Principles  Applied  to  Sensor  Fusion 

The  discussion  of  how  humans  process  information  includes  a  discussion  of  each  of  four  basic 
principles  that  apply  to  human  cognition  and  subsequent  action.  These  principles  are  listed  next 
and  form  the  structure  for  the  discussion  that  follows  regarding  the  specific  elements  of  the  JDL 
model: 

Human  Versus  Machine  Process  Cycles  Comparison 

Human  Processing  of  Information 

Human  Logic  in  the  Processing  of  Information 

Subjective  Factors  That  Affect  Human  Processing  of  Cognitive  Information,  Data  Risk, 
Data  Value,  and  Cognitive  Bias 
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3.1.1  Human  Versus  Machine  Process  Cycles  Comparison 

Although  both  machines  and  humans  work  through  an  input-throughput-output  cycle,  the  processes 
within  that  cycle  are  very  different.  Machines,  as  discussed  in  this  report,  represent  complex  com¬ 
puter-integrated  systems  that  have  the  ability  to  collect  data  from  sources  or  sensors  and  perform 
complex  activities  during  complex  circumstances,  and  such  machines  typically  operate  in  an  objec¬ 
tive,  digital  environment  where  the  flow  of  work  is  controlled  tightly  by  the  processes  inherent  to 
that  machine’s  design  and  programming.  For  example,  a  computer  uses  electricity  generated  by  fuel 
combustion  to  create  digital  data  through  hardware  and  software.  In  contrast,  a  human  metabolizes 
food  to  generate  electricity  to  control  neurons  that  process  infonnation  collected  and  coded  by  the 
senses;  this  information  is  subsequently  organized  by  the  brain  (a  cognitive  process).  Both  systems 
(machine  and  human)  can  yield  a  similar  result  in  many  cases,  but  understanding  each  of  their 
processes  allows  a  beher  understanding  of  the  suitability  of  each  of  those  systems  in  producing  a 
defined  overall  system  output  and  is  also  critical  in  allocating  functions  to  each.  In  the  case  of  many 
examples  described  in  this  report,  a  system  usually  consists  of  human  and  machine  contributions 
toward  a  complex  goal  that  is  ultimately  defined  by  the  human. 

The  human,  as  opposed  to  the  machine,  primarily  functions  in  a  subjective  world  where  all  infor¬ 
mation  and  action  are  controlled  by  numerous,  subjective  processes.  This  huge  difference  in  basic 
operating  system  features  provides  for  the  strengths  and  weaknesses  of  each  system  (human  versus 
machine)  and  the  need  remains  for  both  to  work  together  in  an  integrated  and  complementary 
fashion  to  achieve  goals  that  neither  could  accomplish  alone. 

The  process  styles  mentioned  are  biologically  or  electrically  based,  but  in  processing  machine 
data,  sensor  fusion  systems  use  set  filtering  methods  (e.g.,  Kalman  filters)  that  “keep  updating  the 
system’s  view  of  data  changes  in  a  very  efficient  way  (i.e.,  they  use  the  last  updated  result/data 
point  and  only  use  new  information  to  update  it,  and  then  that  new  data  point  is  later  used  as  the 
reference  point  for  the  next  round  of  new  information)  overall,  which  helps  reduce  the  processing 
requirement  on  the  system.  However,  it  is  critical  that  the  algorithms  are  “human  factored”  so  that 
they  do  not  get  progressively  off  track”  (de  Pontbriand,  2006;  Kalman,  1960;  Kalman  &  Bucy, 
1961). 

Other  types  of  information  filters  include  Markov,  stochastic,  or  Bayesian  approaches  or  even 
neural  nets  where  software  agents  selectively  seek  specific  information  and  update  their  models 
based  on  that  information  (these  machines  not  only  selectively  use  infonnation  sources  but  allow 
the  system  to  tune  itself  through  an  understanding  of  measurement  noise  and  the  use  of  recursive 
processes  where  the  system  can  leam  by  its  mistakes)  (Abdi,  2003;  Pinheiro  &  Lima,  1999). 
However,  many  of  these  algorithms  are  based  on  fixed  sets  of  logic,  whether  it  is  the  consideration 
of  Gaussian  noise  in  the  collection  of  data  (Kalman  filtering)  or  updating  information  based  on 
fixed  algorithms  of  a  set  distribution  (other  Stochastic  methods)  (Pinheiro  &  Lima,  1999). 

In  the  case  of  humans,  the  context  for  these  cognitive  processes  is  often  based  on  not  only  recent 
or  easily  available  data  nor  on  probabilities  but  on  other  factors  such  as  described  by  Klein  as 
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external  knowledge  sources  unavailable  to  machine  systems.  The  recognition-primed  decision 
model  (RPDM)  as  postulated  by  Ross,  Klein,  Thunhohn,  Schmitt,  and  Baxter  (2004)  was  a 
refinement  of  a  model  originally  established  in  1989  by  Klein,  Calderwood,  and  Clinton-Cirocco 
and  describes  the  theory  behind  several  human  cognitive  processes.  The  RPDM  postulates  that 
pattern  matching,  diagnosing  a  situation,  and  evaluating  a  course  of  action  are  all  higher  order 
processes  that  humans  use  in  processing  infonnation  and  solving  problems.  Within  that  three- 
element  structure,  other  factors  such  as  perception  of  a  situation  as  typical  or  atypical  then  leads 
to  a  recognition  involving  expectancies:  relevant  cues,  plausible  goals,  and  typical  actions.  This 
approach  is  quite  removed  from  machine  logic  where  a  more  mechanistic,  deductive  process  is 
often  implemented. 

3.1.2  Human  Processing  of  Information 

One  of  the  processes  illustrated  in  the  JDL  diagrams  is  parallel  processing  of  information  at  several 
levels  (Steinberg,  Bowman,  &  White,  1999;  Hall,  2005).  While  it  is  true  that  humans  can  process 
information  in  parallel,  when  that  task  is  accomplished,  real-time,  overall  task  perfonnance  often 
suffers  dramatically  (Wickens,  1987).  An  analogy  is  the  task  of  driving  while  one  is  talking  on  a 
cell  phone,  an  action  that  can  and  is  accomplished  with  regularity  but  with  significant  performance 
decrements  in  other  tasks.  The  converse  of  this  dual  tasking  problem  is  that  people  can  either 
drive  or  talk  on  a  cell  phone  with  very  good  perfonnance  but  not  in  parallel  unless  they  wish  to 
reduce  their  performance  of  one  or  both  activities  dramatically,  even  though  both  tasks  may  be 
heavily  practiced  (Cohen,  Dunbar,  &  McClelland,  1990).  This  area  of  primary  and  secondary 
tasking  is  also  complicated  by  the  human  sensory  channels  being  used  for  that  processing 
(Wickens,  1987).  However,  it  is  also  possible  for  humans  to  process  information  in  parallel  with 
greater  facility  as  that  activity  becomes  more  automated  via  repetition  and  practice  (Wickens, 

1987;  Cohen,  McClelland,  &  Dunbar,  1990).  Humans  with  a  lot  of  experience  in  a  certain  area  of 
activity  are  called  experts,  and  after  an  activity  is  learned  to  a  high  level  of  expertise,  it  becomes 
more  automated  in  nature.  However,  no  matter  what  the  level  of  human  of  expertise  that  an 
individual  has  in  performing  a  task,  that  task  must  compete  for  limited  time  and  mental  resources 
in  order  to  continue  effectively  (Sanders  &  McConnick,  1993).  Humans  can  operate  to  some 
degree  using  parallel  processing  but  not  nearly  to  the  degree  that  automated,  mechanical  systems 
can  parallel  process  information.  A  specific  model  of  how  humans  process  infonnation  is 
discussed  in  the  following  sections.  The  Wickens’  model  of  attention  (Wickens  &  Hollands, 

2000)  is  shown  in  figure  3. 

This  model  posits  that  our  attentional  resources  (and  by  analogy,  all  our  resources)  are  finite  and 
through  various  mechanisms  associated  with  human  physiology  and  cognition,  address  problems 
through  a  closed  loop  process  involving  sensing  data  (short-term  sensory  store),  perceiving  data  (as 
a  sensor  would  for  a  machine),  and  then  applying  attention  resources  (energy,  motivation, 
cognition,  etc.)  through  an  iterative  process  to  reach  an  outcome  (response).  In  turn,  the  closed 
loop  nature  of  this  action  is  the  feedback  process  called  learning.  This  model  of  human 


6 


performance  is  similar  to  that  of  the  JDL  process  model  but  has  a  wide  variety  of  differences  that 
are  discussed  in  this  report. 
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Figure  3.  The  Wickens'  model  of  human  attention  (Wickens  &  Hollands,  2000). 


Human  cognition  or  the  process  of  thinking  involves  actions  and  outcomes  that  are  currently 
orders  of  magnitude  more  complex  than  those  of  any  machine.  With  a  computer  as  an  example 
of  a  typical  machine,  it  contains  an  ON  and  OFF  control  and  then  an  ever-broadening  array  of 
energy  flow-switching  and  storage  mechanisms  that  allow  digital  processes  to  occur  at  the  speed 
of  light  via  a  binary,  deductive,  and  digital  logic  system.  However,  with  all  that  complexity, 
even  the  most  powerful  computers  would  be  no  match  for  an  infant  in  terms  of  discovery  and 
cognitive  growth.  Without  the  ability  of  locomotion  (inferring  continual  and  dramatically 
varying  sensor  input),  self-sustainment  of  operating  energy,  and  a  diverse  cognition  logic  basis, 
a  machine  cannot  progress  or  learn  at  the  same  rate  as  the  human  in  a  variety  of  domains. 

3.1.3  Human  Logic  in  the  Processing  of  Information 

How  do  these  human-machine  examples  relate  to  the  process  of  sensor  fusion  in  the  formation  of 
conclusions  and  then  actions? 

If  humans  processed  information  purely  in  parallel,  they  would  have  to  consider  all  input  at  the 
same  time  (which  they  cannot  do  since  they  are  perceptually  limited  to  only  a  few  channels  of 
information  processing  [visual,  auditory,  and  proprioceptive]),  and  those  channels  are  bandwidth 
limited.  If  they  were  processing  the  same  items  in  series,  they  would  have  to  consider  each  item 
as  it  contributes  to  the  solution  as  a  sum  (a  very  time-consuming  process).  Neither  approach  is 
regularly  used  to  reach  human-derived  decisions. 

Quite  often,  humans  do  not  use  the  deductive  logic  process  where  premises  follow  directly  to 
conclusions  such  as  the  method  of  formal  proof  that  is  used  in  mathematics  or  that  often  drives 
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common  machine  logic.  Instead,  humans  tend  to  use  a  combination  of  deductive  and  inductive 
logic,  as  well  as  intuition  and  emotion  to  reach  general  conclusions;  however,  if  fonnal  logic  is 
used,  humans  often  use  more  of  the  inductive  process  (called  the  method  of  discovery)  or  infer 
from  the  specific  to  the  general  (Searles,  1956). 

It  is  also  critical  to  note  that  humans  process  all  information  subjectively,  even  when  it  is  objective 
in  nature.  If  posed  with  the  question,  “Will  the  sun  rise  tomorrow?,”  virtually  everyone  would 
indicate  (and  truly  believe)  that  the  sun  will  rise  tomorrow,  but  that  conclusion  is  based  strictly  on 
subjective  information  as  reinforced  by  experience.  Perhaps  one  person  in  a  million  could  fully 
comprehend  celestial  mechanics  and  mathematics  to  the  degree  of  producing  an  objective 
mathematical  proof  to  describe  that  conclusion,  but  even  that  individual  would  have  to  rely  on 
“objective  data”  derived  by  subjective  human  experience  and  intellect.  In  contrast,  machines  are 
strictly  objective  in  terms  of  basic  processes  such  as  the  binary  effects  of  current  being  on  or 
current  being  off  as  the  basis  for  action. 

The  method  of  deductive  logic  is  more  often  used  in  machine  processes  where  conclusions  are 
made  from  the  general  to  the  specific.  With  a  military  example  of  machine  logic,  the  following 
premises  might  be  made  by  an  automated  system:  (a)  a  tank  is  an  object  that  moves;  (b)  a  moving 
metal  object  can  be  detected  by  a  radar  sensor;  (c)  deductively,  an  item  “tank”  can  be  associated 
with  hypotheses  about  what  constitutes  “tank”;  (d)  a  “tank”  object  can  also  be  tracked  or  observed 
by  a  sensor;  (e)  a  machine  can  be  programmed  to  confirm  that  the  object  is  a  “tank”  if  pre-existing, 
deductive  criteria  are  met  (speed,  size,  heat  output,  turning  radius,  etc.).  Given  this  logic  stream, 
that  same  machine  could  then  transfer  a  sensor-driven  pixel  representation  of  a  “tank”  object  to  a 
screen  (in  the  form  of  moving  pixels).  Then  that  movement  of  pixels  can  (with  a  screen  display) 
be  observed  by  a  human  who  might  then  conclude  that  it  was  “a  tank”.  This  same  machine  system 
might  use  elements  of  the  deductive  logic  stream  to  assess  the  “tank”  further  and  provide  “conclu¬ 
sions”  based  on  that  sensor  stream  and  logic  to  conclude  that  the  object  was  a  T-73  tank  or  an  M- 
60  tank. 

In  comparison,  if  a  human  were  observing  that  same  “tank”  image,  s/he  might  analyze  it  for 
familiar  patterns  (pattern  recognition,  pattern  matching)  and  then  form  tentative  hypotheses  and 
create  a  theory  about  the  “object”  tank.  Carrying  this  example  further,  a  human  might  predict  an 
action  performed  by  that  tank  that  cannot  currently  be  accomplished  by  a  machine. 

S/he  would  do  so  using  inductive  logic  processes,  but  s/he  can  also  inductively  extrapolate  the 
concept  and  effects  of  a  “tank”  to  a  much  greater  degree  than  any  machine.  For  example,  a  person 
may  see  the  “tank  object”  and  conclude  that  an  attack  could  result  against  a  friendly  force  at  a 
distance  of  3  kilometers  across  a  small  stream  in  20  minutes — a  conclusion  not  backed  by 
deductive  logic  but  that  may  be  correct  and  was  subjectively  detennined  based  on  experience, 
stored  knowledge,  and  very  dynamic  inductive  processes.  Humans  may  also  mix  and  match 
logical  methods  and  in  doing  so,  do  not  follow  a  formal  logical  pathway  of  pure  deduction  or 
induction.  For  example,  if  the  object  is  a  T-73  tank,  it  could  actually  elicit  a  fear  reaction  from  the 


8 


observer  because  it  is  likely  being  driven  by  an  enemy  crew  with  hostile  inten-tions — a  conclusion 
that  could  be  programmed  into  a  machine  but  only  resulting  in  a  warning  or  caution  and  not  as  the 
actual  emotion  felt  by  the  observer.  The  element  of  fear  induced  by  the  “tank  object”  would  feed 
into  the  information  processing  by  that  human  observer  and  could  even  subjectively  influence  his 
or  her  decision-making  process. 

The  use  of  a  strict  deductive  process  is  not  possible  for  a  human  to  accomplish  because  of  the 
vast  number  of  possible  moving  objects  that  could  be  a  tank,  so  humans  typically  would  not 
deductively  identify  a  moving  object  as  a  tank  versus  a  cow  versus  a  cloud  versus  a  tuning  fork, 
but  they  would  use  an  inductive  process  to  pattern  recognize  and  thus  quickly  detennine  what 
that  object  was  based  on  their  past  experience  (Duda,  Hart,  &  Storck,  2001).  This  type  of 
reaction  to  information  is  also  mirrored  in  the  expectancies  category  of  Klein’s  RPDM 
(discussed  in  section  3.1.1). 

In  contrast  to  human  thinking,  machine  pattern  matching  is  deterministic  and  mathematical  in 
nature  and  is  bound  by  various  set  algorithms  (Haskel,  2007).  It  typically  is  not  capable  of 
learning  and  prioritizing  in  real  time  to  compensate  for  infonnation  loss,  ambiguity,  or  deception. 
As  a  tactical  illustration  of  this  idea,  a  tank  camouflaged  by  white  sheets  and  big  red  balloons 
could  conceivably  halt  or  interfere  with  the  process  of  automated  detection  by  a  machine  because 
that  particular  camouflage  approach  simply  would  not  have  been  chosen  by  a  rational  human  when 
trying  to  deceive  another  human  (based  on  “common  sense”)  and  would  be  a  poor  camouflage 
choice  for  a  human  observer.  However,  a  human  would  quickly  recognize  that  deception  because 
of  his  or  her  ability  to  inductively  prioritize  and  weigh  new  or  unusual  infonnation  in  real  time. 

As  humans,  we  take  this  process  for  granted,  but  machine  logic  cannot,  and  thus  our  abilities  as 
humans  currently  vastly  exceed  machines  in  that  regard.  Our  ability  to  prioritize  and  eliminate 
information  is  as  valuable  as  our  ability  to  collect  and  process  that  information. 

3.1.4  Subjective  Factors  That  Affect  Human  Processing  of  Cognitive  Information,  Data 
Risk,  Data  Value,  and  Cognitive  Bias 

The  previous  paragraphs  have  mentioned  basic  logical  process  differences  between  humans  and 
machines,  but  human  infonnation  processing  contains  considerably  more  complexity  in  how 
humans  process  information.  A  review  of  some  of  those  processes  follows. 

3. 1.4.1  Data  Risk 

A  human  using  a  machine-fused  sensor  system  is  completely  dependent  on  sources  of  information 
that  are  removed  from  his  or  her  own  experience.  If  all  data  were  created  equally,  this  would  not 
present  such  a  problematic  situation,  but  each  bit  of  data  available  to  a  sensor  fusion  process  has  an 
attendant  risk  level  for  humans,  with  some  data  being  quite  a  bit  more  risky  than  other  bits  of  data. 
When  data  are  collected  first  hand  by  the  human,  data  risk  can  be  assessed  based  on  that  human’s 
experience,  but  in  the  remoteness  of  a  situation  where  infonnation  is  far  removed  from  the  source, 
the  credibility  of  those  data  is  usually  not  included  with  the  presentation  of  those  data.  This  is  all 
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too  common  an  issue  when  one  is  processing  printed,  verbal,  or  visual  information.  Each  type  of 
information  has  various  levels  of  face  validity  acceptance,  as  established  by  the  user,  and  that  level 
of  validity  drives  the  acceptance  or  rejection  of  the  data  (Meister,  1985).  A  pilot  tapping  a  linger 
on  a  mechanical  altitude  display  or  a  computer  operator  logging  out  and  then  back  in  are  examples 
of  how  humans  test  information  and  assess  its  risk  level.  Objectively,  a  machine  has  no  conse¬ 
quence  of  risk,  i.e.,  it  cannot  die  from  a  mistake. 

3. 1.4.2  Data  Value 

In  terms  of  information  value,  simply  declaring  something  to  be  “the  way  it  is”  is  low  value  and 
high  risk,  rather  than  a  good  logical  basis  for  action.  That  is,  unsupported  information  is  lower 
value  than  if  that  infonnation  were  presented  with  the  use  of  some  type  of  objective,  externally 
supported  references,  citations,  or  the  concurrence  of  subject  matter  experts  (SMEs). 

Humans  weigh  the  value  of  any  data  using  very  complex  rules,  some  of  which  are  not  consciously 
apparent  to  the  human  making  the  decision.  Processes  of  verification  of  information  involve  not 
only  logic  but  circumstance  and  even  an  understanding  of  the  motivation  someone  may  have  in 
providing  those  data  (Salvendy,  1987).  This  may  also  be  thought  of  in  terms  of  signal  detection 
theory,  where  the  criterion  point  being  set  is  based  on  the  possible  magnitude  of  con-sequences  of 
a  particular  type  of  occurrence.  This  weighing  of  data  is  important  for  any  subjective  decision,  and 
one  way  in  which  this  type  of  data  could  be  viewed  is  from  the  perspective  of  bias. 

3. 1.4. 3  Cognitive  Data  Biasing 

The  term  “cognitive  bias”  implies  a  conscious  or  unconscious  attempt  to  weigh  an  outcome 
according  to  some  type  of  pre-existing  rule  or  criterion.  Bias  may  or  may  not  cloud  a  person’s 
judgment  in  regard  to  making  decisions.  The  issue  is  that  cognitive  bias  cannot  be  easily 
dismissed  and  can  influence  human  judgments  independently  of  facts. 

Cognitive  bias  is  one  of  the  factors  that  control  human  actions  to  some  degree,  and  in  this  regard, 
we  are  very  different  from  a  machine  that  does  not  operate  under  that  type  of  subjective  condition. 
Our  sensory  systems  are  all  susceptible  to  unconscious  and  uncontrollable  biases,  and  we  call 
certain  of  these  perceptual  biases  “illusions”.  Most  everyone  is  familiar  with  optical  illusions  in 
which  objects  do  not  appear  to  represent  reality,  depending  on  perspective  (figure  4).  Simply 
knowing  that  such  biases  exist  does  not  make  them  go  away;  they  are  present  even  in  the  face 
of  our  logical  knowledge  that  tells  us  what  infonnation  we  think  should  be  presented.  As  an 
example,  when  we  view  an  object  such  as  a  set  of  railroad  tracks  moving  away  to  the  horizon,  our 
visual  sense  limitations  would  lead  us  to  believe  that  the  rails  converge  at  a  distance,  but  logically, 
we  know  that  they  do  not.  As  such,  we  must  also  recognize  that  humans  are  biased  not  just  in  the 
perceptual  sense  but  in  the  cognitive  processes  that  we  employ  to  collect  and  process  data.  The 
work  of  Kahneman  and  Treisman  (1984)  covers  a  number  of  cognitive  biases  as  well  as  ways  that 
humans  process  information  through  rules  of  thumb,  technically  known  as  heuristics.  A  heuristic 
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is  a  pattern  of  thinking  which  we  apply  cognitively,  in  much  the  same  way  as  Klein  and  his  RPD 
model  would  describe  as  an  expectancy  or  plausible  goal. 


Figure  4.  Two  types  of  human  bias  (illustrations  drawn  from  public 
domain  world-wide  web  images). 

Conversely,  a  bias  can  also  be  a  result  of  learned  behavior,  which  empowers  us  to  make  a  good 
decision  with  a  moderate  or  even  low  level  of  information  being  present.  For  example,  the  sound 
of  squealing  tires  immediately  attracts  attention  because  of  our  prior  experience  and  a  bias  toward 
associating  that  precursor  sound  with  an  impending  vehicle  accident  (figure  4).  That  potential 
accident  may  or  may  not  occur,  but  our  previous  experience  and  a  bias  to  trust  our  experience 
allows  a  quick,  often  life-saving  decision  to  be  made  in  the  absence  of  additional  information. 

Like  the  startle  response  to  a  loud  noise,  our  action  is  involuntary  but  is  evolutionarily  adaptive, 
and  with  the  RPDM  and  the  previous  example,  the  sound  of  squealing  tires  is  not  only  a  relevant 
cue  to  action  but  an  expectancy  of  things  to  come. 

The  human  user  of  sensor  systems  data  often  can  conclude  outcomes  based  not  on  vast  amounts  of 
stored  information  like  a  computer  but  on  critical,  linked  memories  that  indicate  an  overall  trend. 
This  is  a  critical  function  of  human  cognition  in  that  we  quite  often  make  judgments  based  only  on 
fragments  of  information,  and  sometimes  those  judgments  are  correct. 
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4.  Results 


4.1  The  Five  Levels  of  the  JDL  Model  of  Sensor  Fusion 

Each  of  the  sensor  fusion  levels  has  characteristics,  and  each  of  those,  in  turn,  has  human  factors 
implications.  The  following  sections  discuss  the  human  factors  of  human  interaction  with  sensor 
processing.  The  JDL  Fusion  Model  levels  are 

•  Level  0  -  Pre-processing 

•  Level  1  -  Object  Refinement 

•  Level  2  -  Situation  Refinement 

•  Level  3  -  Threat  Refinement 

•  Level  4  -  Process  Refinement 

•  Level  5  -  User  Refinement 

4.1.1  JDL  Level  0,  Pre-processing:  Signal  or  Feature  States 

This  function  is  largely  out  of  the  realm  of  human  engineering  and  is  one  of  machine  design  at  the 
hardware  and  software  level.  At  this  stage,  the  organization  and  identification  of  data  are  based 
more  on  machine  capabilities  than  on  human  limitations  or  capabilities.  The  ability  to  detect  a 
target  within  a  forest  or  the  range  of  a  sensor  is  driven  more  by  physics  than  by  human  psycho¬ 
physics.  However,  the  mechanical  features  of  any  sensor  or  data  collection  system  should  consider 
the  capabilities  and  limitations  of  human  users  and  should  address  those  inherent  values  in  their 
design  specifications  and  data  products  so  as  not  to  overload  the  human  who  receives  that  infor¬ 
mation  stream.  Sensor  systems  should  output  their  information  in  ways  to  maximize  human 
recognition  and  processing. 

The  collection  and  use  of  data  of  any  type  should  consider  the  ultimate  user  of  those  data,  and  the 
product  of  that  process  should  match  the  abilities  of  its  user.  For  example,  if  an  electro-optical 
machine  can  scan  500  square  miles  of  territory  per  minute,  it  must  store  that  infonnation  and 
provide  it  to  an  image  analyst  in  a  fonnat  s/he  can  understand  and  comprehend,  and  most  likely, 
this  would  not  be  at  the  rate  that  it  was  originally  collected.  However,  if  the  purpose  of  that  scan 
was  to  identify  any  large  troop  movements  within  that  area,  as  defined  by  a  pre-set  moving  target 
identification  (MTI)  algorithm,  then  that  type  of  troop  movement  could  be  automatically  isolated 
from  the  less  valuable  mass  of  data  and  transmitted  to  the  analyst  in  seconds,  thereby  allowing  him 
or  her  to  make  a  critical  detennination  of  intent  without  ever  having  to  view  all  the  collected  data. 
In  this  sense,  pre-processing  and  sensor  fusion  have  refined  the  data,  converted  them  to  useful 
human  information,  and  communicated  that  information  to  an  analyst  in  a  form  that  s/he  can 
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understand  and  interpret  in  an  order  of  magnitude  less  time  than  if  s/he  were  forced  to  observe  the 
entire  stream  of  information. 

One  of  the  biggest  interface  mismatches  at  Level  O  processing  is  between  what  a  machine  can 
generate  and  what  a  human  can  absorb.  Time,  as  well  as  the  quantity  and  quality  of  information, 
must  be  matched  so  that  information  is  fed  to  the  human  at  a  rate  that  the  human  can  actually 
use. 

4,1.2  JDL  Level  1,  Object  Refinement:  Entity  Parametric  and  Attributive  States 

This  section  is  the  core  step  of  where  the  JDL  model  directly  interacts  with  the  human  infonnation 
refinement  processes.  In  order  to  refine,  define,  or  organize  objects,  human  perception  is  the  first 
step  of  that  process.  Human  perception  steps  as  described  by  Foley  and  Moray  (1987)  are  dis¬ 
cussed  in  terms  of  the  perception  of  information  brought  to  us  by  our  senses  and  include  infonna¬ 
tion  about  sight,  hearing,  rotation,  falling  and  rectilinear  movement,  taste,  smell,  touch,  vibration, 
pressure,  temperature,  cutaneous  pain,  position  and  movement  (kinesthesia).  Unfortunately,  very 
few  of  these  sensory  faculties  are  available  to  help  us  make  decisions  in  a  typical  sensor  fusion 
environment,  which  probably  provides  only  visual  imagery  and  perhaps  sound  information.  For 
example,  an  unmanned  aircraft  system  (UAS)  operator  may  only  see  a  video  screen  image  trans¬ 
mitted  by  his  or  her  UAS,  with  no  aircraft  sound,  no  movement,  and  no  other  sensory  feedback 
from  that  system  other  than  the  visual  channel.  In  comparison  to  actual  experience,  the  informa¬ 
tion  contained  in  that  control  environment  is  very  limited  and  superficial. 

In  contrast,  if  that  same  operator  were  able  to  be  in  that  UAS,  s/he  would  see  what  it  sees,  hear  its 
operation  noise,  smell  the  UAS  exhaust,  the  engine  vibration,  observe  the  firing  of  a  missile  and 
its  effects,  and  would  thus  build  an  integrated  and  richer  mental  model  of  the  concept  “UAS”. 
However,  in  the  typical  sensor  fusion  operations  environment,  nearly  all  those  sources  of  infor¬ 
mation  about  the  UAS  are  not  available.  The  object  UAS  becomes  abstracted  when  presented  by 
filtering  and  a  single  channel  presentation  (video  screen).  This  principle  of  abstractness  has  often 
led  to  a  disconnection  with  the  object  being  observed  remotely  and  is  seen  as  a  problem  with  work 
stations  in  UAS  or  other  robotic  systems  (Cooke,  Pringle,  Pedersen,  &  Connor,  2006). 

All  sensor  systems,  whether  they  are  machine  or  human,  create  multiple  levels  of  abstraction 
since  they  represent  reality  but  are  not  thu  reality  (Ceruti,  2004).  Most  information  systems, 
including  sensor  fusion  systems,  would  be  more  effective  if  they  were  more  immersive  in  nature 
and  provided  complementary  sources  of  information  that  mirrored  reality.  In  effect,  the  results 
would  be  less  abstract  and  less  symbolic.  If  a  UAS  operator  could  actually  be  on  board  a  UAS, 
his  or  her  awareness  of  that  system  would  be  enhanced,  but  that  connectedness  to  the  UAS  is  by 
definition  impossible  to  achieve.  Therefore,  the  only  way  to  approximate  that  state  is  for  the 
operator  to  have  an  interface  that  duplicates  the  more  important  system  performance  features  that 
might  be  found  within  and  about  that  UAS.  The  following  object  refinement  human  processes 
are  discussed  in  relation  to  level  1  JDL  fusion  and  are  believed  to  be  critical  to  proper  human 
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understanding  of  a  sensor-fused  or  other  mechanical  system,  as  well  as  being  critical  to  the 
cognitive  processing  of  any  information. 

4. 1 .2. 1  The  Speed- Accuracy  Trade-off 

The  speed-accuracy  trade-off  is  easily  summarized  since  actions  can  be  fast  or  accurate  but  not 
extreme  in  both  directions  at  the  same  time  (Sanders  &  McCormick,  1993).  Skilled  performance 
of  an  expert  can  be  fast  and  accurate  in  comparison  to  that  of  a  novice,  but  even  within  that  expert, 
there  is  always  a  trade-off  between  these  competing  factors  in  a  conflicting  and  competing 
resource  allocation  process  (Proctor  &  VanZandt,  1994). 

4. 1.2.2  Top-Down  Processing 

Top-down  processing  is  a  series  of  human  cognitive  processes  that  involve  the  processing  and 
refinement  of  information  (Klapuri,  2007;  Parasuraman,  2006).  These  processes  include  the 
Gestalt  process,  perceptual  stereotypes,  signal  detection,  and  attention  (Wickens,  1987).  A 
discussion  of  Gestalt  processes,  stereotypes,  and  signal  detection  follows. 

4. 1.2. 3  Gestalt  Processes 

Gestalt  processes  refer  to  the  human  ability  to  see  patterns  in  data  or  imagery  that  intuitively  seem 
to  fit,  without  detailed  analysis  or  thought  (Koffka,  1935).  These  are  a  form  of  pattern  recognition 
that  is  quick  and  fairly  effortless.  In  principle,  Gestalt  perception  is  often  associated  with  visual 
processes,  but  it  is  not  just  confined  to  the  visual  sense. 

Gestalt  principles  are  a  very  powerful  tool  to  quickly  discern  patterns  in  information.  For  example, 
in  the  scene  in  figure  5,  the  Gestalt  perspective  quickly  identifies  objects  that  do  not  “fit”  in  the 
picture,  so  both  the  unusual  conically  shaped  tree  and  a  concealed  tank  are  visible  at  a  glance 
because  of  pattern  recognition  and  matching.  The  “Gestalt”  of  this  picture  is  that  of  two  objects 
that  do  not  “fit”  into  the  background  and  thus  are  quickly  identified  as  something  other  than 
background. 


Figure  5.  Gestalt  perception  (Image  from  public  domain  world-wide  web). 
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Stereotypes  are  learned  stimuli  that  can  allow  an  observer  to  define  objects  quicker  than  if  the 
information  associated  with  those  objects  were  not  coded  in  this  manner.  This  is  because  they 
have  a  certain  compatibility,  agreed  to  by  a  large  portion  of  a  population  (Bullinger,  Kem, 
Muntzinger,  in  Salvendy,  1987). 

Common  stereotypes  include  color  coding  (green  =  good,  red  =  hot,  red  force  =  enemy,  blue  force  = 
allies).  This  type  of  perceptual  coding  system  (Proctor  &  VanZandt,  1994)  is  used  widely  within  the 
Army  and  is  typical  in  the  coding  of  force  types  (cavalry,  artillery,  tanks,  platoon,  battalion,  etc.). 

Stereotypes  often  have  no  inherent  information  but  rely  on  memorization  in  order  to  achieve  their 
meaning;  however,  a  positive  trend  in  symbolic  stereotyping  is  to  use  objects  (icons)  that  resemble 
the  objects  that  they  portray,  so  for  example,  a  tank  icon  would  be  displayed  with  a  track,  a  basic 
tank  chassis  outline,  and  a  gun  barrel,  each  bit  of  information  contributing  to  the  concept  “tank,” 
and  to  a  limit,  this  provides  faster  and  more  accurate  detection  and  discrimination  by  the  observer. 

Stereotypes  also  have  limitations  in  the  amount  of  information  they  can  convey  and  they  primarily 
focus  on  quick,  not  necessarily  accurate,  discriminations.  As  figure  6  shows,  many  military  symbols 
do  not  have  realistic  (iconic)  qualities;  thus,  they  must  be  memorized  for  content.  However,  some  of 
the  symbols  do  have  iconic  qualities  (e.g.,  helicopter  blades  or  the  outline  of  a  tank  [armor]). 


Figure  6.  Stereotypes  and  iconic  symbols  (image  from  the  Federation  of 
American  Scientists,  2006). 
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4. 1.2.4  Signal  Detection  Theory 

The  term  “signal  detection  theory”  was  first  used  in  electronic  signal  communication  in  the  1950s 
and  subsequently  was  adopted  and  expanded  greatly  by  psychologists  to  describe  the  perception  of 
a  signal  in  the  presence  of  noise  (Shannon,  1948).  For  psychologists,  the  term  “signal”  has  been 
broadened  to  tasking,  goals,  or  perception,  while  the  term  “noise”  indicates  any  factor,  environ¬ 
mental  or  other,  that  interferes  with  that  process. 

This  concept  of  signal  detection  is  typically  diagrammed  with  the  use  of  two  normal  curves.  One 
curve  represents  the  signal  (goal,  task,  perceived  object)  and  the  other  curve  represents  noise  or 
anything  that  interferes  with  that  process  of  perception;  the  intersection  of  the  two  curves  repre¬ 
sents  the  signal-to-noise  relationship  for  a  circumstance.  If  the  curves  are  widely  separated  (lower 
part  of  figure  7),  there  is  little  interference  with  the  task  or  goal,  but  as  the  noise  curve  further 
intrudes  on  the  task  or  goal,  the  greater  the  level  of  masking  and  thus,  the  greater  difficulty  of 
performing  the  task  or  meeting  the  goal  (illustrated  by  the  upper  part  of  figure  7).  When  the  two 
curves  overlap  completely,  the  goal  or  task  is  not  possible.  This  concept  is  a  powerful  analog  for 
human  performance  in  terms  of  human  workload. 


4. 1.2. 5  Attention  and  Vigilance 

Humans,  unlike  machines,  do  not  operate  on  a  set  or  binary  type  of  scale  (that  is,  on  or  off).  As  a 
system,  they  operate  through  a  variety  of  operating  continua,  one  of  which  is  called  attention.  For 
any  task,  a  person’s  involvement  with  that  task  is  driven  by  his  or  her  level  of  attention.  Generally, 
poor  attention  equals  poor  performance,  proper  attention  equals  proper  performance,  and  over¬ 
attention  can  lead  to  a  cognitive  myopia  or  attentional  tunneling  (excessive  focus  on  too  small  an 
area). 

The  presentation  of  any  data  involves  time,  and  one  area  that  humans  are  not  noted  for  is  their 
ability  to  sustain  vigilance  for  extended  periods  of  time.  For  example,  during  World  War  II,  sonar 
operators  were  shown  by  Mackworth  to  have  greatly  diminished  vigilance  after  only  30  minutes  of 
operation,  often  missing  vital  signals  because  of  boredom  and  a  vigilance  decrement  (Parasuraman, 
2007).  UAS  operators  often  are  confronted  by  long  periods  of  inactivity  followed  by  frantic  action, 
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followed  again  by  inactivity,  and  this  work  environment  is  not  conducive  to  maintaining  vigilance. 
Performance  decrements  have  been  modeled  and  reductions  in  effectiveness  predicted  (Cooke  et  ah, 
2006),  and  there  is  a  large  body  of  research  that  concerns  the  loss  of  performance  with  sustained 
vigilance.  A  sample  of  research  showing  a  short-term  vigilance  decrement  is  shown  in  figure  8  and 
is  called  the  Mackworth  (1948)  vigilance  curve.  This  curve  involved  the  results  of  detecting  changes 
in  the  ticking  of  a  clock  over  time.  Essentially,  it  showed  that  for  boring,  repetitious,  or  infrequent 
tasks,  performance  suffers  greatly. 

Contrary  to  vigilance  decrement,  sensor  fusion  system  users  must  be  presented  with  information 
for  a  time  adequately  long  enough  to  process  that  infonnation  (minimal  response  time  limits). 
Continuity  of  information  flow  at  the  right  pace  is  critical  in  matching  machine  feeds  of  infor¬ 
mation  and  tasking  to  the  abilities  of  the  user. 


Figure  8.  Mackworth  (1948)  vigilance  curve  (human  perfonnance 
decrement  over  time). 

4.1.3  JDL  Level  2,  Situation  Refinement 

After  the  perceptual  elements  have  been  addressed  by  the  analyst  at  Level  1  processing,  situation 
refinement  begins  to  become  more  dominant  at  the  JDL  Level  2  (situation  refinement  level).  At 
this  stage,  the  human  processes  of  cognition  (rather  than  just  perception)  become  more  involved. 

It  is  also  important  to  note  that  there  is  no  clear  defining  line  between  human  perception  and 
cognition,  and  perceptual  biases  can  easily  cross  over  into  cognitive  processes  and  conclusions. 

Raw  data  from  external  sources  are  pre-processed  by  mechanical  systems  in  Level  0,  perceptually 
arranged  and  manipulated  in  Level  1,  and  are  then  evaluated  in  Level  2.  As  an  example  from  a 
military  intelligence  analyst  environment,  the  six  functions  associated  with  this  level  are  (Blasch  & 
Plano,  2003) 

1 .  Aggregation  of  information  into  entities  or  force  structures, 

2.  Determination  of  relationships  and  working  rules, 

3.  Interpretation  of  the  actions  of  entities  of  forces, 
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4.  Determination  of  the  overall  purpose  of  large  and  small  actions, 

5.  Hypotheses  of  current  and  future  activities  and  testing  those  hypotheses, 

6.  Resolution  of  anomalous  factors  and  the  consideration  of  deception. 

Each  of  the  six  processes  within  Level  2  builds  an  awareness  of  a  situation  and  does  so  iteratively 
as  information  is  brought  together,  tested,  and  evaluated.  There  is  never  actually  an  end  product 
for  situation  awareness  (SA);  it  is  an  ongoing,  dynamic  process  that  continues  even  after  the  reso¬ 
lution  of  a  particular  situation. 

Before  discussing  the  six  individual  processes  of  Level  2,  it  would  make  sense  to  define  SA  since 
it  is  an  awareness  of  infonnation  in  the  first  place  that  then  proceeds  to  all  further  processes. 

Dr.  Mica  Endsley’s  (1988)  definition  of  SA  is  the  commonly  accepted  origin  of  that  tenn: 

The  perception  of  the  elements  in  the  environment  within  a  volume  of  time  and  space,  the 
comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future. 

The  user  of  a  sensor  fusion  system  must  perceive  information  that  is  dependent  on  that  system’s 
perceptual  abilities  and  limitations.  S/he  must  be  able  to  comprehend  that  information  (thus,  much 
of  the  discussion  here  about  the  processes  of  comprehension  and  cognition),  and  then  s/he  must  be 
able  to  project  or  predict  a  possible  outcome,  based  on  that  perception  and  comprehension.  Both 
SA  and  sensor  fusion  systems  have  a  similar  closed  loop  process  and  use  an  input-throughput- 
output  cycle  that  is  familiar  to  industrial  engineers  and  human  factors  practitioners.  Considering 
this  overall  perspective  of  SA,  the  following  steps  of  the  JDL  level  2  model  are  discussed  in  terms 
of  human  factors  principles. 

4. 1.3.1  JDL  Level  2,  Step  1 :  Aggregation  of  Human-Related  Data 

Of  the  six  processes,  the  process  of  aggregation  is  readily  suitable  for  integration  with  current 
automated  systems.  The  ability  to  collect  data  and  detennine  correlations  or  separate  data  into 
categories  is  most  easily  done  by  automated  means  as  directed  by  the  human.  Lor  example,  given 
a  large  collection  of  enemy  combatants  and  the  goal  of  finding  a  key  person(s)  from  that  database, 
a  database  sorting  program  can  immediately  sort  those  players  into  categories  as  defined  by  a 
military  intelligence  analyst.  Lactors  associated  with  geography,  age,  birth  place,  combatant 
service  affiliation,  notable  activities  that  the  person  has  been  involved  with,  places  where  s/he  has 
lived,  personnel  s/he  has  been  in  contact  with,  education  or  schooling  affiliations,  friends,  family 
lineage,  military  specialty,  and  a  wide  variety  of  known  variables  can  all  be  sorted  and  organized 
by  an  automated  routine.  Aggregation  could  also  apply  to  combination  of  sensor  data  from  a 
variety  of  sources,  the  prioritization  of  those  data,  and  the  use  of  those  data  to  build  a  model  or 
reach  a  conclusion. 

4. 1.3. 2  JDL  Level  2,  Step  2:  Determination:  Human  Relationships  and  Attributes 

Determination  of  relationships,  attributes,  and  working  rules  is  an  area  where  automation  could 
provide  help  to  the  human  in  the  loop.  With  the  military  intelligence  domain  as  an  example, 
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enemy  personnel  relationships  can  be  provided  by  the  correlation  of  information  (such  as  which 
enemy  personnel  might  be  related  to  whom),  which  set  of  personnel  might  be  associated  with 
another  set  of  personnel  in  a  chain  of  command,  or  a  general  understanding  of  the  psychological 
or  sociological  working  rules  for  determining  such  enemy  activities.  Nevertheless,  machine -based 
associative  algorithms  for  enemy  personnel  matching  could  provide  clues  to  human  analysts  of 
correlations  not  obvious  from  the  mere  collection  of  human  intelligence  (HUMINT)  in  the  field. 

4. 1.3. 3  JDL  Level  2,  Step  3:  Interpretation 

This  sub-process  seeks  to  answer  the  question  why  was  an  action  taken,  and  it  often  involves 
human  subjective  data.  It  seeks  to  answer  questions  such  as  “What  role  did  this  action  have?”  or 
“What  effect  would  an  action  like  this  have  on  some  defined  process?”  It  is  attempting  to  derive 
meaning  from  actions.  It  is  almost  entirely  a  cognitive  process  that  is  aided  little  by  current 
automated  systems.  One  way  in  which  an  automated  system  might  be  able  to  contribute  to 
answering  these  types  of  questions  could  be  in  the  graphic  presentation  of  magnitude  of  effects 
or  associations  with  the  use  of  pie  charts  or  graphs  that  show  trends.  By  compiling  numeric  data 
and  reporting  them  in  a  graphic  way,  an  analyst  can  improve  his  or  her  overall  understanding. 

4. 1.3. 4  JDL  Level  2,  Step  4:  Determination:  Large  and  Small  Actions  and  Their  Effects 

This  process  is  like  construction  of  a  jigsaw  puzzle  with  the  analyst  placing  piece  after  piece  of  the 
puzzle  together  to  assess  its  impact  on  the  overall  picture.  Not  all  information  is  created  equally; 
some  provides  higher  information  content  than  others.  Human  beings  have  a  fairly  good  ability  to 
generalize  from  small  pieces  of  information  and  to  determine  a  sense  of  proportion  of  the  value  of 
information.  This  intuition  process  is  based  on  past  knowledge  and  often  can  manifest  itself  in 
conclusions  with  only  a  small  amount  of  infonnation  being  required;  however,  it  can  also  lead  to 
quick  conclusions  not  supported  by  facts. 

4. 1.3. 5  JDL  Level  2,  Step  5:  Hypotheses:  Testing  of  Human-Related  Information 

The  formation  of  hypotheses  by  the  sensor  fusion  infonnation  user  is  perhaps  one  of  the  most 
important  cognitive  tasks  that  s/he  must  perform,  and  that  testing  is  often  performed  with 
subjective  data.  Often,  a  checklist  or  matrix  approach  can  be  used  by  the  analyst  to  evaluate 
(but  not  prove)  a  hypothesis.  In  many  cases,  intuition  alone  is  the  basis  for  decision,  whereas 
some  more  formal  and  structured  method  of  testing  hypotheses  should  be  encouraged. 

4. 1.3. 6  JDL  Level  2,  Step  6:  Resolution:  Situational  Human  Factors 

This  process  often  involves  unusual  cases  where  ambiguity  is  very  high.  If  a  situation  does  not 
appear  to  make  sense  or  fit  an  established  pattern,  it  very  well  may  be  some  act  of  deception 
accomplished  deliberately  to  raise  doubt,  promote  confusion,  or  preempt  the  use  of  resources  in 
its  investigation.  This  aspect  of  sensor  fusion  has  no  machine  analog  since  machines  have  not  yet 
been  created  to  lie,  subjectively  obscure,  or  in  other  ways  purposefully  distort  infonnation.  The 
creation  of  decoys,  setting  up  false  troop  movements,  creation  of  meaningless  activity,  propaganda 
releases,  hacking,  and  other  forms  of  deception  are  often  challenges  to  military  sensor  fusion 
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systems  users  because  they  impugn,  distort,  or  destroy  the  quality  of  information  that  the  sensor 
fusion  system  is  transmitting. 

4.1.4  Level  3,  Fusion:  Threat  Refinement,  Estimation,  Prediction,  and  Utility 

This  stage  of  the  sensor  fusion  process  begins  to  isolate  causes,  to  predict  outcomes,  and  to  assess 
the  utility  of  those  actions.  At  this  stage  of  sensor  fusion,  the  human  and  the  machine  are  taxed 
with  interpolating  data,  creating  predictive  outcomes,  and  then  determining  if  those  predictions 
have  utility  in  comparison  to  the  real-world  events  they  are  modeling.  The  third  step  of  SA, 
“prediction,”  is  critical  in  this  stage. 

4.1.5  Level  4,  Fusion:  Process  Refinement  and  Human  Learning 

Reviewing  the  sensor  fusion  process  from  start  to  finish  is  also  a  learning  process  and  involves 
training.  Since  training  is  also  a  cognitive  process,  psychologists  or  human  factors  experts  should 
be  consulted  when  a  training  plan  or  system  is  being  created.  Improvement  and  optimization  of 
any  process  is  the  domain  of  industrial  engineering,  but  the  psychological  elements  of  learning  are 
taught  as  basics  to  human  factors  practitioners.  The  learning  process  should  always  be  closed 
loop,  that  is,  iterative  and  repetitive  in  order  to  continually  improve  a  product  or  a  process. 

4.1.6  Level  5,  Fusion:  User  Refinement 

This  process  should  involve  far  more  than  visualization,  but  it  is  currently  associated  with 
visualizing  enhancements  of  the  other  four  levels.  This  stage  also  reflects  the  ability  of  the 
user  to  customize  or  tailor  the  process  used  to  achieve  the  sensor  fusion  system  goals.  It  must 
be  remembered  that  human  beings  have  five  sensory  channels,  each  of  which  provides  informa¬ 
tion  to  support  cognitive  activities.  If  a  system  interface  is  limited  only  to  visual  processes,  it  is 
inherently  disconnected  with  how  humans  naturally  process  information.  It  is  understandable  that 
much  of  our  information  is  acquired  visually  as  we  sit  in  front  of  a  computer  screen  and  observe 
data  products;  however,  our  ability  to  learn  in  the  real  world  is  greatly  enhanced  by  our  using  the 
rest  of  our  sensory  channels  to  process  information.  The  following  elements  could  be  added  to 
tools  for  user  enhancement  beyond  the  most  common  visual  channel  and  primarily  reflect  future 
technological  or  human  engineering  approaches  to  fusing  data  from  complex  sources  such  as 
HUMINT. 

4. 1 .6. 1  Auditory  Feedback  and  Learning  Human-Machine  Comparisons 

The  human  auditory  channel  is  a  rich,  high-density  communication  medium.  In  the  analysis  of 
speech  alone,  many  linguistic  principles  can  be  used  in  providing  feedback  and  learning  through 
sensor  fusion.  However,  linguistics  alone  is  a  very  complex  study  and  not  currently  very  successful 
in  terms  of  machine  processing,  other  than  simple  word-for-word  or  phrase  translator  systems.  In 
contrast,  human  observers  using  logic  and  experience  can  extract  meaning  from  text  or  spoken 
language  which  is  veiled  by  semantic  features  such  as  sarcasm  or  formalism  that  contain  a  very 
great  amount  of  information  but  would  not  be  detected  by  machine  translation  systems.  Psycholo- 
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gists  understand  that  inflection,  the  choice  of  words,  and  the  manner  in  which  they  are  delivered  all 
contain  huge  amounts  of  information  that  are  lost  when  those  words  are  merely  written.  In  this 
respect,  human  perception  of  auditory  feedback,  when  understood  in  the  context  of  linguistic  and 
semantic  rules,  can  provide  a  human  with  far  better  information  conclusions  than  a  machine  can. 

4. 1.6.2  Sociological  and  Biodynamic  Factors 

As  humans  who  live  in  a  sociological  setting,  posture,  gesture,  physical  movement,  and  internal 
bio-physiological  factors  can  tell  us  a  lot  about  human  behavior.  Body  language  is  a  colloquial 
tenn  for  an  externally  driven,  sociologically  influenced  communication  process.  It  is  entirely 
possible  that  a  sensor-fused  system  could  even  provide  cues  from  posturing,  gestures,  or  move¬ 
ments  that  could  indicate  critical  details  such  as  deception  and  denial.  In  terms  of  human  percep¬ 
tion  and  cognition,  simply  observing  an  enemy’s  spoken  language  over  time  may  contribute  to  a 
sociological  perspective  that  helps  an  analyst  better  in  fusing  complex  information.  In  the  past, 
he  detectors  were  used  to  assess,  from  a  physiological  perspective,  the  presence  of  deceptive 
techniques  with  the  use  of  only  one  type  of  electro-physiological  measurement  (electromyogra¬ 
phy).  In  the  future,  powerful,  automated  routines  could  be  developed  to  assess  deception  and 
denial  by  combinations  of  bio-analytical  routines  involving  eye  movement,  posture,  limb  move¬ 
ment,  or  physiological  measures  such  as  electro-cardiogram,  electro-encephalography,  or  electro¬ 
myography. 


5.  Summary  Discussion 


This  report  has  briefly  covered  the  essential  elements  of  the  JDL  definition  of  sensor  fusion  and 
discussed  how,  for  each  of  those  elements,  human  factors  can  contribute  valuable  human-centered 
information  to  those  processes  and  products  of  sensor  fusion.  The  infonnation  conveyed  in  this 
report  was  presented  to  demonstrate  that  human  engineering  principles,  the  principles  of  human 
psychology,  cognition,  learning,  and  an  understanding  of  human  behavior  can  all  help  create  a 
better  method  of  fusing  data  into  useful  information. 


6.  Future  Directions 


The  tasking  demands  required  for  many  military  operations  are  bound  to  become  more,  rather  than 
less,  complex  over  time.  Asymmetric,  small-scale  war  operations  in  a  network-centric  environ¬ 
ment  provide  personnel  with  the  risk  of  being  lost  in  a  sea  of  data,  while  having  to  anticipate  an 
enemy’s  action  that  is  not  traceable  by  simple  observation  of  battalions  of  tanks  or  thousands  of 
troops  en  masse.  In  addition,  being  removed  from  direct,  experiential  contact  with  the  enemy  and 
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the  ability  to  employ  all  of  one’s  senses  in  assessing  infonnation  can  result  in  superficial  judg¬ 
ments  based  on  infonnation  that  is  brittle  in  context,  rather  than  rich  in  content.  In  addition,  the 
current  warfare  environment  still  has  huge  risks  associated  from  enemy  combatants  that  do  not 
follow  many  of  the  historic  rules  of  warfare,  nor  do  they  provide  traditional  trails  of  infonnation. 
In  order  to  combat  this  trend,  technology  and  sensor-fused  systems  have  been  implemented  to 
compensate  for  this  change  in  direction. 

Planning  for  conventional  military  operations  against  conventional  military  forces  may  not  be  the 
common  battle  environment  for  this  generation,  but  that  process  must  adapt  to  this  new  environ¬ 
ment  in  order  to  maintain  the  upper  hand  in  maintaining  capable  military  strength.  The  use  of 
sensor-fused  systems  has  only  now  begun  to  emerge  from  a  beginning  in  rudimentary  tracking 
algorithms  to  the  concept  that  sensor  fusion  may  provide  a  very  powerful  tool  in  synthesizing  and 
simplifying  information.  This  fusion  process  may  have  more  inherent  value  than  matching  large 
numbers  of  conventional  military  forces  against  each  other  in  a  well-defined  tactical  environment. 
If  knowledge  is  power,  sensor  fusion  may  be  a  powerful  tool  in  future  military  operations. 
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Acronyms 


F-22 

HFE 

HUMINT 

JDL 

MTI 

RPDM 

SA 

SME 

SRI 

UAS 


Fighter,  Model  22,  U.S.  Air  Force 
human  factors  engineering 
human  intelligence 
Joint  Directors  of  Laboratories 
moving  target  indication 
recognition-primed  decision  model 
situation  awareness 
subject  matter  expert 
not  an  acronym 

unmanned  aircraft  system  (formerly  called  an  unmanned  aerial  vehicle  [UAV]) 
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